In previous reports, [3H]5-HT has been used to characterize the pharmacology of the rat and human 5-HT2B receptors. 5-HT, the native agonist for the 5-HT2B receptor, has a limitation in its usefulness as a radioligand since it is difficult to study the agonist low-affinity state of a G protein-coupled receptor using an agonist radioligand. When using [3H]5-HT as a radioligand, rauwolscine was determined to have relatively high affinity for the human receptor (Ki human = 14.3+/-1.2 nM, compared to Ki rat = 35.8+/-3.8 nM). Since no known high affinity antagonist was available as a radioligand, these studies were performed to characterize [3H]rauwolscine as a radioligand for the cloned human 5-HT2B receptor expressed in AV12 cells. When [3H]rauwolscine was initially tested for its usefulness as a radioligand, complex competition curves were obtained. After testing several alpha2-adrenergic ligands, it was determined that there was a component of [3H]rauwolscine binding in the AV12 cell that was due to the presence of an endogenous alpha2-adrenergic receptor. The alpha2-adrenergic ligand efaroxan was found to block [3H]rauwolscine binding to the alpha2-adrenergic receptor without significantly affecting binding to the 5-HT2B receptor and was therefore included in all subsequent studies. In saturation studies at 37 degrees C, [3H]rauwolscine labeled a single population of binding sites, Kd = 3.75+/-0.23 nM. In simultaneous experiments using identical tissue samples, [3H]rauwolscine labeled 783+/-10 fmol of 5-HT2B receptors/mg of protein, as compared to 733+/-14 fmol of 5-HT2B receptors/mg of protein for [3H]5-HT binding. At 0 degrees C, where the conditions for [3H]5-HT binding should label mostly the agonist high affinity state of the human 5-HT2B receptor, [3H]rauwolscine (Bmax = 951+/-136 fmol/mg), again labeled significantly more receptors than [3H]5-HT (Bmax = 615+/-34 fmol/mg). The affinity of [3H]rauwolscine for the human 5-HT2B receptor at 0 degrees C did not change, Kd = 4.93+/-1.27 nM, while that for [3H]5-HT increased greatly (Kd at 37 degrees C = 7.76+/-1.06 nM; Kd at 0 degrees C = 0.0735+/-0.0081 nM). When using [3H]rauwolscine as the radioligand, competition curves for antagonist structures modeled to a single binding site, while agonist competition typically resulted in curves that best fit a two site binding model. In addition, many of the compounds with antagonist structures displayed higher affinity for the 5-HT2B receptor when [3H]rauwolscine was the radioligand. Typically, approximately 85% of [3H]rauwolscine binding was specific binding. These studies display the usefulness of [3H]rauwolscine as an antagonist radioligand for the cloned human 5-HT2B receptor. This should provide a good tool for the study of both the agonist high- and low-affinity states of the human cloned 5-HT2B receptor.